User-State Labeling Procedures For The Multimodal Data Collection Of SmartKom

This contribution deals with the user-state labeling procedures of a multimodal data corpus that is created in the SmartKom project. The goal of the SmartKom project is the development of an intelligent computer-user interface that allows almost natural communication with an adaptive and self-explanatory machine. The system does not only allow input in the form of natural speech but also in the form of gestures. Additionally, facial expressions are analyzed. For the training of recognizers and the exploration of how users interact with the system, data is collected. The data comprises video and audio recordings from which the speech is transliterated and gestures and user-states are labeled. This paper gives an in depth description of the different annotation procedures for user-states. Some preliminary results will be presented, particularly a description of the homogeneity of the different user-states and their most important features.